Applying Practice to Theory 

Ryan Williams* 



55 ' Abstract 

; 

I How can complexity theory and algorithms benefit from practical advances in computing? 

We give a short overview of some prior work using practical computing to attack problems in 
^ , computational complexity and algorithms, informally describe how linear program solvers may 

O ■ be used to help prove new lower bounds for satisfiability, and suggest a research program for 

I developing new understanding in circuit complexity. 

a^ : 

I — ^! 1 Introduction 
U; 

. As hardware becomes steadily more powerful, computer scientists should repeatedly ask themselves 

c/3 I what can we do with all the spare computing power at our disposal? There have been many inventive 

1 and exciting answers to this question in the form of distributed computing projects, from the study 
of protein folding (Folding@Home) to the improvement of climate models (Climateprediction.net) 

! to the search for extraterrestrial intelligence (SETI@Home) to the mathematically intriguing (the 

I Great Internet Mersenne Prime Search). 

■ I would like to suggest that some spare computing power should be devoted towards achiev- 

, ing a better understanding of computation itself: solving significant open problems in theoretical 

' ' computer science. I would hope that this suggestion is indisputable. Considering all that we do 

I ■ not understand in the theory of computing, exploiting extraneous computation for improving ba- 
sic knowledge should be a priority. An attractive property of this suggestion is its potential for 

^P. I self- improvement. Ideally, improvements in basic knowledge can lead to more spare cycles in the 

' future, leading to further advances in basic knowledge, and so on. 

. While the above looks splendid in writing, of course it is not clear how to better the causes of 



theory in this way. In fact this problem may be exceptionally difficult in most cases of interest. 
My hope in this survey to encourage readers to think more seriously about the problem. Proofs 
reliant on computer calculation have become increasingly more common in mathematics. Examples 
include the proof of the four color theorem by Appel and Haken [AHTTa^ IAH77b| which has been 
greatly simplified |RSST97] . Hales' proof of the Kepler Conjecture [HalOSj . Hass and Schlafly's 
proof of the double bubble conjecture |HSOO| . Lam's proof that there is no finite projective plane of 
order 10 |LTS89j . and McCune's proof of the Robbins Conjecture |McC97) . In general, computers 
can play a much greater role than simply discovering or verifying proofs, the end products of 
mathematical research. To prove theorems we need to formalize a proof system, and in doing so 
we may have to finagle messy details. Computers are possibly more useful in helping us discover 
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new computational artifacts suggestive of a deeper paradigm, such as an ingeniously tiny circuit 
for 7x7 matrix multiplication which inspires us to look for something better. 

At this point I should note a distinction between two usages of computers in proofs. For 
convenience, I will designate them infeasibly checkable and feasibly checkable. In the former, a 
computer is needed to generate the proof of the theorem, and the "proof" is the code of some 
program, the trace of that program's execution, and our observation that the program output the 
appropriate answers. I personally have no problem with such proofs so long as they are carefully 
reviewed, but acknowledge that they stretch the bounds of what one typically calls a proof. In 
contrast, feasibly checkable proofs may require strenuous computing effort, but once obtained they 
can be checked by mathematicians within a reasonable time frame. Restricting ourselves to feasibly 
checkable proofs keeps the task of proof- finding roughly within NP, and I would like to strongly 
promote this usage of computers whenever possible. 

First I will survey a few ideas in this vein that have been introduced in algorithms and complexity 
theory. Then I will discuss some recent work using computers to find proofs of time lower bounds 
in restricted models and small circuits. I will not give prescriptions for solving your major open 
problems via distributed computing with spare desktop cycles. But I do wish that this article helps 
you consider the possibility. 

2 Some Prior Work 

Let me begin by saying that I cannot hope to cover all the innovative uses of computers in algorithms 
and complexity. My goal is to merely point out a few theory topics that I know where practical 
computing has made a noteworthy impact. 

2.1 Moderately Exponential Algorithms 

In the area of moderately exponential algorithms, the goal is to develop faster algorithms for solving 
NP-hard problems exactly. Of course we do not expect these faster algorithms to be remotely 
close to polynomial time. Instead we settle for exponential algorithms that are still significant 
improvements over exhaustive search. For example, a 0(1.9" •poly(m)) time algorithm for Boolean 
Circuit Satisfiability on n variables and m gates would be very interesting, but not necessarily for 
any practical reason. The ability to avoid brute-force search in such a general case would be an 
amazing discovery in itself. Currently we have no idea how to construct such an algorithm, or even 
if its existence would imply something unexpected. However, we can solve the 3-SAT problem in 
0(1.33") time with a randomized algorithm, by taking advantage of the structure of 3-CNF. This 
is due to Iwama and Tamaki |IT04] and is just the latest of a long line of results on the problem. 

2.1.1 Analyzing Exponential Algorithms 

Typically, one of the central problems in an exponential algorithms paper is to prove good upper 
bounds on the running time of some algorithm which solves some hard problem. (Usually the 
correctness of the algorithm is straightforward.) Much effort has been undertaken to better un- 
derstand the behavior of backtracking algorithms. To aid the discussion, let us work with a toy 
example. For a node ?; in a graph, let N{v) denote the set of v's neighbors. Consider the following 
algorithm for solving Minimum Vertex Cover: 
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// all nodes have degree at most two, solve in polytime. If a node has degree 0, remove 
it from the graph and recurse on the remaining graph. If a node has degree 1, then remove 
it and its neighbor u, recurse on the resulting graph getting a cover C , and return C U {u}. 
Otherwise take a node v of highest degree. Recurse on the graph with v removed, getting a 
cover Ci. Recurse on the graph with v and N{v) removed, getting a cover C2. Return the 
minimum of Ci U {v} and C2 U N(v). 

We won't explain how to solve the degree-two case here, but leave it to the reader. How do we 
analyze the runtime of such an algorithm? A natural approach is to try writing a recurrence. Let 
T(n) be the runtime on an n node graph. In the two recursive calls of the algorithm, we remove 
at least one node and at least four nodes, respectively. (We remove at least four nodes because 
|-^(^^)| ^ 3.) This gives us the recurrence 

r(n) < T{n - 1) + T(n - 4) + poly(n). 

Solving this recurrence in the usual way (by finding a real root of /(x) = 1 — 1/x — 1/x^) we find 
that T(n) < 1.39". While exponential, this is still better than the obvious 2" algorithm, since we 
can now handle instances of more than double the size in the same running time. 

Our analysis certainly has much slack. One can imagine cases where the number of nodes 
removed increases by much more, and we have not taken them into account. More generally, it is 
not clear that n is the best "progress measure" for the algorithm. Perhaps if we count the number 
of edges in the graph instead, we may find a better runtime bound for sparse graphs. Indeed, in 
the first recursive call, at least three edges are removed (since we chose a v of largest degree) and 
in the second call we remove at least five edges: at least three edges for the neighbors of v, and at 
least two additional edges since the neighbors have degree at least two. We have 

T{m) < T{m - 3) + T{m - 5) + 0(poly(n)), 

leading to T[m) < 0(1.19™). For sufficiently sparse graphs, this improves 1.39". More ambitiously, 
we could try to capture both observations with the double recurrence 

T(m, n) < T{m — 3, n — 1) + T(m — 5, n — 4) + 0(poly(n)). 

Now how do we deal with this? One way is to convert the double recurrence into a single one, by 
letting k = aim + a2n. Then 

T{k) < T{k - 3ai - 02) + T{k - 5ai - 4a2) + 0{po\y{k)). 

So T{k) < 0{c^) where c is a real solution to 1 — i/x^'^i^^^ — iy'a;5a!i+3a2 — q_ -poi an example when 
CKi = 1/2 and 02 = 1 then the runtime bound is 0(1.21'"/^'^"'). In this way, we can interpolate 
between the two time bounds. 

In general, optimizing this sort of analysis can become terribly complicated. When there are 
many possible cases in the algorithm, and different variable measures are decreasing at different 
rates, the analysis becomes intractable to carry out by hand. However, researchers have found ways 
to apply computers to the problem. Eppstein |Epp06| showed that multivariate recurrences similar 
to the above can be approximately solved efficiently, by expressing the problem as a quasi-convex 
program. This has become a very useful tool. For example, we could keep track of the number of 
nodes nj of degree i in its time recurrence expressions, for i > 2 (and for sufficiently large k, we 
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lump the number of degree > k nodes into a single quantity n>k)- This can also be converted into 
a single variable recurrence, introducing weights for each n^. 

For some algorithms we can get surprisingly good time bounds in terms of the total number of 
nodes: quasiconvex optimization uncovers interesting a^'s. Intuitively, this makes sense, because 
removing high degree nodes should contribute more to the progress of the algorithm than removing 
those of low degree. Fomin, Grandoni, and Kratsch [FGKOSal IFGK06) have found 0(1.52") and 
0(1.23") algorithms for Dominating Set and Maximum Independent Set/Minimum Vertex Cover 
respectively, by performing analyses of the above kind on simple new algorithms, using Eppstein's 
computer approach to determine optimal settings of aj. For more details, see the survey [FGKOSb] . 
Scott and Sorkin [SS07] have analyzed algorithms for Max 2-CSP and related graph problems 
with a somewhat similar approach, keeping track of the degrees of neighbors in the recurrence 
as well. Their approach formulates the analysis with linear programming instead. Provided the 
original recurrences are reasonably sized, the above approaches can generate feasibly checkable 
proofs: after the optimization has found appropriate weights, one can often manually check that 
the recurrence works out. 

2.1.2 Case Analysis of Exponential Algorithms 

Another approach is to have a computer check that a recursive algorithm will admit an efficient 
time recurrence, over all the possible inputs up to a certain size. Since many recursive backtracking 
algorithms work very locally (they only look at a subgraph of finite size around a specially chosen 
node) this sort of case analysis is sometimes enough to ensure a good upper bound on the running 
time. However, this style of approach typically does not lead to feasibly checkable proofs of upper 
bounds, and the case analysis is done by computer. 

Robson [RobOlj . in an unpublished technical report, has written a program to do this for 
a Maximum Independent Set algorithm, proving that the algorithm runs in 0(2"/"^) time. In 
particular, a lengthy and very complex extension of the above toy algorithm is presented and 
analyzed case-by-case, using a computer to enumerate many of the possible cases. 

In the toy algorithm, there were some special cases prior to backtracking: removing nodes of 
degree less than two, and solving instances with only degree two nodes. Rules similar to degree 
and degree 1 node removal are called simplification rules. In general, these are short polytime 
rules that allow one to reduce the size of an instance practically for free, provided that a certain 
substructure exists. Fedin, Kojevnikov, and Kulikov |FK061 IKK06) developed a natural formalism 
for expressing special cases in SAT problems, which made it not only possible for a computer to 
perform case analyses, but also search for new simplification rules for MAX SAT and SAT on its 
own, resulting in faster new algorithms. For earlier work of this kind, see [NSOSj IGGIIN04] . 

Could it be possible to search over (or reason about) all recursive backtracking algorithms in 
some sense, and show exponential limitations on solving problems like SAT? Here I am using "all" 
very loosely; without clever simplification rules, exponential lower bounds on treelike resolution 
already give lower bounds on simple backtracking. Perhaps there is a 2^"" algorithm for every 
e > 0, by using a sufficiently complicated backtracking algorithm. Many seem to disbelieve in 
this possibility; some work has articulated this belief, in some sense. For instance, Alekhnovich 
et al. [ABBIMPOS] formalized a model for backtracking algorithms, proving that 3SAT requires 
2^(") time in their model. However their proof uses a SAT instance that encodes a linear system 
of equations over GF(2), which can be solved trivially in polynomial time. Such results teach us 
that we should not be myopically focused on one specific algorithmic technique. 
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2.2 Approximability and Inapproximability 

Since the early 90's there has been significant progress in the study of hard-to-approximate prob- 
lems, aided by the celebrated PCP theorem |AS98t [A.LMSS98] . An algorithm is a p-approximation 
for a minimization problem 11 if on all inputs the algorithm outputs a solution that has value at 
most p times the minimum value of any solution. In the maximization case, the output solution 
must have cost at least p times the maximum. Note when 11 is a minimization (maximization) 
problem, we have /O > 1 (p < 1), respectively. A prime objective in the study of approximability is 
to determine for which p a problem can be p-approximated in polytime, and for which p a problem 
is NP-hard to p-approximate. Several surprisingly tight results are known; for instance, a random 
assignment satisfies at least 7/8 of the clauses in any 3-CNF formula (with three distinct variables 
in each clause) , yet Hastad showed [HasOl] that it is N P-hard to satisfy a number of clauses that is 
at least 7/8 -|- e of the optimum, for any e > 0. That is, a polytime (7/8 -|- e)-approximation would 
imply P = NP. 

Here I will briefly survey a couple of works in the study of approximation that rely heavily on 
computer power to achieve their results. 

2.2.1 Gadgets via Computer 

Everyone who has seen an N P-completeness reduction knows what a gadget is. Treating one problem 
n as a programming language, you try to express pieces of an instance of H' by constructing gadgets, 
simple components that can be used over and over to express instances of 11' as instances of H. 

To illustrate, consider the standard reduction from 3-SAT to MAX 2-SAT due to Garey, John- 
son, and Stockmeyer |GJS76| . One can transform any 3-CNF formula F into a 2-CNF formula F' , 
by replacing each clause of F such as Cj = (^i V £2 V £3), where £1, 12-, and £3 are literals, with the 
"gadget" of 2-CNF clauses 

(£1), {I2), (^3), {yt), Hi V -^2), V -£3), Hi V -4), {h V -^vi), {h v -y.), [h v -y^), 

where yi is a new variable. If an assignment satisfies q, then exactly 7 of the 10 clauses in the 
gadget can be satisfied by setting yi appropriately. If an assignment does not satisfy q , then exactly 
6 of the 10 can be satisfied. Therefore F is satisfiable if and only if 7/10 of the clauses in F' can 
be satisfied. This reduction also says something about the approximability of MAX 3-SAT: 

Proposition 2.1 If an algorithm is a {1 — e)- approximation for MAX 2-SAT, then by applying the 
reduction one can obtain a [1 — 7 e)- approximation to MAX 3-SAT^ 

Recall we mentioned that MAX 3-SAT does not have a polytime (7/8 -|- e)-approximation unless 
P = NP. Hence the proposition implies that MAX 2-SAT cannot be (55/56 -|- e)-approximated 
in polytime unless P = NP. So gadgets can be used to extend inapproximability results from one 
problem to another^ 

How good is the above gadget from 3-SAT to 2-SAT? Could we find a gadget that implies 
stronger inapproximability for MAX 2-SAT? To address these kinds of questions, Trevisan, Sorkin, 
Sudan, and Williamson [TSSWOO] formahzed gadgets, following |BGS98| : 

^To see this, let be the number of clauses in the original 3-CNF F, let < rrig be the optimal number of 
clauses that can be satisfied in F, and let fns be the number of clauses in F satisfied by running the MAX 2-SAT 
approximation on F' and translating the output back to an assignment on the variables of F. By our assumption we 
have (1 — e) < g™^^^» ■ By algebraic manipulation and the fact that 7713 < ma we derive that ^ > 1 — 7e. 

^Note that the best known inapproximability result for MAX 2-SAT uses a different gadget reduction, cf. [HasOlj . 
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Definition 2.2 Let a,£,n > 1, let f : {0,1}'^ {Oil}; '^''^d let T he a family of functions from 
{0, 1}'^+" to {0, 1}. An a-gadget reducing f to J- is given by a set of auxiliary variables yi, . . . ,y„ 
and weights Wj > coupled with constraints Cj G J-, where j = 1, . . . ,i. For every a G {0, l}'^, 

• // /(a) = 1 then (V6 G {0, 1}") J2j WjCj{a, b) < a and {3b G {0, 1}") J2j WjCj(a, h) = a. 

• // /(a) = then (V6 G {0, 1}") Y.j WjCj{a, 6) < a - 1. 

Note it is fine to place weigtits on constraints: to "unweight" them, we can simply make a 
number of copies of each constraint in the instance, proportional to the weights. Observe the 
reduction from 3-SAT to MAX 2-SAT is a 7-gadget reducing /(xi,X2,X3) = xi V 2:2 V 2:3 to the 
family of functions representable by 2-variable clauses, where n = 1, i = 10, and wj = 1 for all j. 

First, [TSSWOO] showed that if we fix the number of auxiliary variables n, then the requirements 
in the gadget definition can be described by a large number of linear programs, with a large 

number of inequalities in each linear program. That is, by specifying n and a tuple (Ci, . . . , Ce) G 
J-^, the problem of setting Wj^s to minimize a and satisfy the gadget definition boils down to solving 
a large linear program which has inequalities dealing with every possible a G {0, l}'^'. This is almost 
obvious, except that the definition has elements of a 0-1 integer program: when /(a) = 1, we must 
ensure that there is an assignment b making the sum equal a. To circumvent this, [TSSWOO] also 
try all possible functions B from the set of satisfying assignments of / to {0, l}". Then, for every 
a such that f{a) = 1, we simply set b = B{a) in the WjCj{a, b) = a" constraints of the linear 
program. As one might expect, this can lead to some very large linear programs, but for small 
constraint functions they are manageable. 

Still, we had to fix n to get a finite search space, and it is entirely possible that gadgets keep 
improving as n increases. ^TSSWO O] prove that, for satisfying very natural conditions, it suffices 
to set n < 2*, where s is the number of satisfying assignments of /. These conditions are satisfied 
by 2-CNF and many other well-studied constraint families. 

Using the computer search for gadgets, the authors proved several interesting results in ap- 
proximation which are still the best known to date; for example, (16/17 + e)-approximating MAX 
CUT is NP-hard. Incidentally, their search also uncovered an optimal 3.5-gadget reducing 3-SAT 
to MAX 2-SAT: taking a as before, the 2-CNF gadget is 

(£1 V ^3), i^h V -^3), (^1 V -^Vi), (-£1 V Vi), ih V -2/), (-^3 V y), {£2 V y), 

where the weights are 1/2 for every clause, except for the last one which has weight 1. In the 
unweighted case, this amounts to having two copies of the last clause. The results here are feasibly 
checkable, for small constraints with a small number of auxiliary variables. 



2.2.2 Analyzing Approximation Algorithms 

Earlier, we noted that any 3-CNF formula can be approximated within 7/8 by choosing a random 
assignment. But if some of the clauses are 2-CNF or 1-CNF, this no longer holds. However, it 
would be strange if we could not 7/8-approximate general MAX 3-SAT because of this. Karloff and 
Zwick |KZ97] proposed a possible 7/8-approximation based on semidefinite programming (SDP)ll 

''For the purposes of this article, just think of semidefinite programming as a generahzation of hnear programming 
where the inequalities are between linear combinations of inner products of unknown vectors, and the task is to find 
vectors satisfying the inequalities. Such systems are approximately solvable in polynomial time. 
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Their algorithm is a fairly direct translation of MAX 3-SAT to an SDP, similar to the MAX CUT 
algorithm of Goemans and Williamson |GW95] . where a vector Vi in the solution corresponds to 
the variable rcj in the formula, and one vector vt corresponds to TRUE. Given the vectors returned 
by the SDP solver, one obtains an assignment to the formula by picking a random hyperplane that 
passes through the origin and setting Xi to TRUE if and only if Vi and vt lie on different sides of the 
hyperplane. Such a hyperplane can be chosen by choosing a normal vector r uniformly at random 
from the unit sphere in R". 

Analyzing the Karloff-Zwick algorithm is very difficult. In order to prove that the algorithm 
is a 7/8-approximation, one needs to prove sharp bounds on the probability that four vectors 
Vi,Vj,Vk,Vt from the SDP lie on the same side of a random hyperplane (i.e., the probability that 
the clause (xj Vxj Vx^) is falsified by the random assignment). This amounts to proving bounds on 
the volume of certain objects whose corners are chosen uniformly at random from the unit sphere, 
which Karloff and Zwick call "volume inequalities." 

Karloff and Zwick were unable to prove strong enough inequalities to get a 7/8-approximation, 
but they did obtain some partial results and gave a conjectured inequality that, if true, would imply 
that the algorithm is a 7/8-approximation. Zwick [Zwi02] proved this inequality along with others 
by writing a program that used interval arithmetic. Interval arithmetic is a method for computing 
over real numbers on a computer in a controlled way, so that all errors are accounted for. The 
proofs of the Kepler conjecture and double-bubble conjecture mentioned earlier also utilize interval 
arithmetic in a critical way. 

In interval arithmetic, one represents a real number r by an interval [ro,ri] where ro,ri are 
machine representable and < r < ri. Ideally, one wants tq (ri) to be as large (small) as possible. 
For a real number r, let r = ri and r = rg. We define 

[ro,ri] + [so,si] = [ rp + so ,ri + gjj, 

[ro,ri] ■ [so,si] = |"min{ rogo , rp , si , ri , gp , ri , gj } , maxjfp^, r^T^T, nTsp, nTsi}] ■ 

One can define more complicated functions similarly. The point is that by doing numerical com- 
putations in interval arithmetic, the resulting interval must contain the correct value, even if that 
value cannot be machine represented. 

But how can we use interval arithmetic to prove an inequality? The key step in Zwick's work 
is a technical reduction from the desired volume inequality to the task of proving that a certain 
system of constraints has no solution over the reals. (Most of these constraints are inequalities, but 
some are disjunctions of inequalities.) Zwick then wrote a program, called RealSearch, which 
takes any system of constraints over bound variables of the form 

/i(xi, . . . ,a;„) > 0, . . . ,/fc(xi, . . . ,x„) > V /fe+i(xi, . . . , a;„) > 0, . . . , 

and tries to prove that they have no solution. Let the bounds on Xi be ai < Xi < bi for some machine 
representable aj,6j. The program starts by letting the interval Xi = [ai,bi] to denote Xj, and 
evaluates the /-functions with interval arithmetic. If any of the constraints fail on this assignment 
(e.g., fi{Xi, . . . , Xn) < 0), then the system returns no solution. Otherwise, the program breaks 
some Xi into subintervals X- = [a,, (oj + bi)/2], X'l = [{oi + 6i)/2, and recursively tries to verify 
that the system of constraints fails with both Xi := X[ and Xi := X'l . Of course, such a procedure 
may not terminate, so the program is instructed to quit after some time. But surprisingly, this 
simple program can verify the necessary inequality, as well as several others that arise in SDP 



7 



approximations! Even though we are not working exphcitly over the reals, if we find that the 
system of constraints fails over all appropriately chosen intervals, then it follows that the system 
fails over all reals. In principle, this strategy could work for any functions / definable in interval 
arithmetic. 

Of course, the resulting proof of the volume inequality is of the infeasibly checkable variety, 
relying on the correctness of the program and the correctness of the floating-point operations. 
Even greater issues arise with Tom Hales' proof of the Kepler conjecture, which requires that his 
programs be run on a processor that strictly conforms to the IEEE-754 floating point standard. 
However, Zwick's strategy does not require that much stringency, and I believe it should be better 
known as a general method for attacking difficult inequalities. 

3 Time Lower Bounds 

I have recently found a nice domain in complexity theory where computer searches help perform the 
"hard work" in the proofs of theorems: namely, in proving time lower bounds for hard problems such 
as SAT on restricted computational models. In this case, the computer generates feasibly checkable 
proofs of lower bounds. Since this style of time lower bounds has been surveyed thoroughly by Van 
Melkebeek |vM04t IvMOTj . I will not provide substantial background here. Instead I will focus more 
on describing how the reduction to a computer search works. For more details, please consult the 
draft available [Wilns] . 

All the lower bounds amenable to computer search have one unifying property: the restricted 
model in which a lower bound is proved can be simulated asymptotically faster on an alternating 
machine. We call such a phenomenon a speed-up property. This property is crucial for the arguments 
to work. Here we will work with time lower bounds for SAT on random access machines that use 
only n"^^) workspace. Define DTS[t(n)] to be the class of problems solvable by such machines in 
t{n) > n time (the acronym stands for "deterministic time with small space"). Here is one example 
of the speed-up property in this setting. 

Theorem 3.1 DTS[t(n)] C Z2TIME[t(n)i/2n"(i)] n n2TIME[t(n)i/2^°(^)]. 

(The classes 5I2TIME and n2TII\/lE are defined in the usual way.) That is, we can simulate 
a small space computation with a square-root speedup using alternations. The proof is due to 
Kannan [Kan84] . but the basic idea goes back to Savitch |Sav70j . The idea is to guess snapshots 
of the n"*^^) space algorithm at t^^"^ points during its computation, then verify in parallel that the 
guesses are correct. Given an algorithm A that runs in time t and uses space n°'^^^ , the corresponding 
5I2 algorithm B{x) existentially writes t^^^ configurations Co,...,C^i/2 of A{x), where Co is the 
initial configuration of A{x) and C^i/2 is an accepting configuration. Since A uses only n°^^^ space, 
these configurations can be written down with n°^^^ bits each. Next, B{x) universally writes 
i G {0,... - 1} and jumps to the configuration Cj. Then it simulates A{x) from Ci for i^/^ 
steps, accepting if and only if A{x) is in configuration Cj+i. The n2-simulation can be defined 
analogously. 

Theorem 13.11 is already enough to prove a non-trivial lower bound for SAT, after applying a 
few more observations from the literature. The first observation is that if SAT is in DTS[n'^], then 
NTIME[n] C DTS[n'^-poly(logn)]. This follows from the fact that SAT is very strongly NP-complete: 
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Theorem 3.2 ([Coo88l ISch78^ IFLvMVOS] ) For every L G NTIME[n], there is a reduction from 
L to SAT that maps strings of length n to formulas of size npoly(logn), where an arbitrary bit of 
the reduction can be computed in poly(log n) time. 

The proof is a very technical version of Cook's theorem which we wih not describe here, but 
let us note in passing that other problems such as Vertex Cover also enjoy a similar property. By 
padding and the fact that DTS classes are closed under complement, we have the following. 

Theorem 3.3 If SAT is in D J Sin"], then for all k andt{n) >n, TkT\ME[t{n)] C Zfe_iTIME[t(n'=)] 
and nfcTIME[t(n)] C nfc_iTIME[t(n^)]. 

Theorem 13.31 says we can remove alternations from a computation, with a small slowdown in 
runtime. (For this reason, I like to call it a "slow-down theorem.") Theorem 13.11 says we can add 
alternations to a DTS computation, with a speedup in running time. Naturally, one's inclination 
is to pit these two results against one another and see what we can derive. Assuming SAT is in 
DTS[n'^], we find 

NTIMEfn^] C DTSln'^"] C ZsTIMEK] C NTIMEK'], 

where the first and third containments follow from Theorem 13. 3|. and the second containment 
follows from Theorem 13.11 When c < 2^/^, the above contradicts the nondeterministic time hierar- 
chy [Coo72] . We have proved the following: 

Theorem 3.4 ( [FLvM V05j ) SAT cannot be solved by an algorithm that runs in n^^^ time and 
72°(i) space, for every e > 0. 

With a more complicated argument involving the same tools, |FLvMV05] proved that SAT 
cannot be in DTS[n'^~'^], where cj) = 1.618 ... is the golden ratio. We can prove a simple n^'^ lower 
bound by generalizing Theorem 13.11 

At this point it will be helpful to introduce some new notation, and this notational shift is crucial 
for the automated approach. Letting t{n) be a polynomial and letting h > (logt(n))/(logn), define 
the class (3 t{n))^C to be the class of problems solvable by a machine that existentially guesses 
t{n) bits, then selects 0{n^) of those bits (along with the input) and feeds them as input to a 
representative machine from class C. (The selection procedure is required to take only linear time 
and logarithmic space, so it does not interfere with any of the time/space constraints of the class.) 
We define (V t{n))^C similarly. By properties of nondeterminism and co-nondeterminism, note that 
we can "combine" adjacent quantifiers in a class: 

Proposition 3.5 (3 ti(n))^^(3 t2{n))^^C = (3 ti{n) + t2{n))^^C, and the analogous statement with 
V also holds. 

Theorem 13 . 1 1 can now be stated more generally: 

Theorem 3.6 (Speedup Rule) For all x such that n < < t{n), 

DTS[i(n)] C (3 n^+"(i))^(V \ogxfDJS[t{n)/n'']. 

The theorem also holds when we interchange V and 3. 
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Theorem 13.61 holds because we can just guess + 1 configurations (instead of f(n)^/^ + 1 as 
before), universally pick i G {0, . . . ,n''}, and the input to the final DTS computation will simply 
be the original input along with the pair of configurations (Cj,Cj+i), which has size n°^^\ Hence 
we have n^+''(^) in the 3-quantifier, and 0(n) bits of input to the final DTS class. 

Our new notation also lets us to state the "slow-down theorem" in a more precise way. 

Theorem 3.7 (Slowdown Rule) // SAT is in DTS[n'^] then for all ai,bi, . . ., a^, b^, Ofe+i > 1, 
and Qi £ {3,V}, the class {Qi n'^^f^ ■ ■ ■ {Qk-i n''^-^f^-^{Qk n"'=)*'=DTS[n'''=+i] is contained in the 
class {Qi n"!)'" • • • (Qfc-i n"fe-i)'''=-i DTS[n^-™^^^^*-i'"'='"'=+i}+°(i)]. 

Again, the result holds by Theorem 13.21 and a standard padding argument. In particular, 
''''-i(Qfc n'''=)^feDTS[n"''-+i] is contained in either NTIME[n'^^'''t'''=-i''''='"fe+i^] or coNTIME[n™'''''t'''=-i'"'='"''-+i^], 
and both of these are in DTS[n'^'™'^^^^*=-i'"'''''^*+i^+°(^)]. We are now ready to prove a stronger time 
lower bound for SAT. 

Theorem 3.8 SAT cannot he solved by an algorithm running in n^'^ time and n°^^^ space. 

Proof. Suppose SAT G DTS[n^] where \/2 < c < 1.6. By TheoremES NTIME[n] C DTS[n^+°(i)]. 
We can derive NTIME[n'^/^+^/'^] C NTIME[n'^' /^+°(^^], ignoring o(l) factors for simplicity: 

NTIME[n^/2+2/c] ^ DTSK'/2+2] (Slowdown) 



C 

c 

c 
c 
c 

c 



3 n"'/=^)'='/2(V logn)iDTS[n2] (Speedup, with x = 

3 n"'/2)c2/2(y iogn)i(V n)\3 logn)iDTS[n] (Speedup, with x = 1) 

3 n^^/^yV^^y n)^{3 log ri)iDTS[n] (Proposition [32]) 

3 n^'/2)c2/2(y ^)iDTSK] (Slowdown) 

3 n"'/2)c2/2Djs[^c2j (Slowdown) 

3 n"'/2)(3 ^cV2)cV2(y iogn)'='/2DTS[n^'/2] (Speedup, x = 
3 n'^'/^f/^y logn)^'/2Djs[^cV2] (Proposition ESD 
3 n^'/2)c2/2Djs[^c3/2] (Slowdown) 

c3/2l 



C NTIME[n 

When c/2 + 2/c > c^/2 (which happens for c < i±f^ ^ 1.6004) we have a contradiction to 
the nondeterministic time hierarchy. □ 

The best known SAT time lower bound (with space) is from an earlier paper of ours |Wil07] . 
and it achieves n'^'^°^i'^/'^) > n^-^. It is a fairly elaborate inductive proof that builds on the same 
ideas. A natural question is if we can do even better than this. It is clear that we have a very 
specific type of proof system on our hands; it is also powerful, in that all known time-space lower 
bounds for SAT (and QBF) on random access machines work over it. 

There is nothing too complicated about the proof of Theorem 13. St we are just applying the 
Speedup and Slowdown Rules in clever ways. More interestingly, the proof was discovered by a 
computer program. Furthermore it is the best lower bound one can prove with only 7 applications 
of Speedup and Slowdown Rules, and we know this because a computer program tried all the cases. 

How did it try the cases? It would seem that the space of possibilities is too high: how could a 
computer try all possible expressions for the exponents? One can show that once we have specified 
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the sequence in which the Speedup and Slowdown Rules are applied, the task of finding the optimal 
lower bound argument can be formulated as a linear program. This makes our job of finding good 
lower bound proofs much easier. 

Let me sketch how a hnear program can be constructed, for a fixed sequence of rules to apply. 
One can show that the sequence of rules completely determines the number of quantifiers in each 
class in the chain of inclusions of the proof. (There are actually two ways to apply the Speedup 
Rule, one where we introduce a ^2 computation and the other a r\2 computation, but we can prove 
that one of these applications is always superior.) Suppose we have a sequence of inclusions such as 
those in the proof of Theorem l3.8l but all exponents in the polynomials are replaced with variables. 
So for example, the second inclusion (or "line") in the proof of Theorem 13.81 becomes 

(3 n''2>3)fc2,2(v j^a2,2)fe2,iDTS[n'*2,i]^ 

the initial class NTIMEfn'^/^+^/^J is replaced with NTIME[n"0'i], and the final class NTIME[n'''^/2] 
becomes NTIME[?7-"*'i]. In general, we replace the exponents in the ith line with variables Oj.j, bij. 
Now we want to write a linear program in terms of these variables that expresses the applications 
of the two rules and captures the fact that we want a contradiction. To do the latter is very easy: 
we simply require 03,1 < ao,ii or 03,1 < ao,i — e for some e > 0. To express a Speedup Rule on the 
ith line, we introduce a parameter Xj > and include the following inequalities: 

Oi,! > 1, aj_i > Ui-i^i — Xi, bi^i = ai,2 = 1, bi,2 > Xi, 6j_2 > h-i,!, ai,3 > ai~i,2, 

ai,3 >Xi, (y k : 4 < A; < m) a^^fc = ai_i,fc_i, (V/c: A < k < m - I) bi^k = h~i,k~i- 

Intuitively, these constraints express that the class {Qm n"™)^™"^ • • •''^ {Q2 n"^)^! DTS[n"i] on the 
(i-l)th line is replaced with {Qm n^^f--^ ■ ■ -^^ {Q2 n"^ax{a2,x,}jmax{a'„6i}^g^ ^■^b,Qj^^mi,^{a,-x,,i}^ 

on the ith line, where Qi is the quantifier opposite to Q2- (One can check that this indeed simulates 
the Speedup Rule faithfully.) We can express the Slowdown Rule in a similar way, by treating the 
desired lower bound exponent c as a constant. Note if we minimize the sum of Uij + bij over all 
i,j, then the above inequalities faithfully simulate max. 

Given that we can take a sequence of rules and turn it into an LP that can then be solved, 
what sequences are good to try? The number of sequences to search can be reduced by establishing 
several properties of the proof system that hold without loss of generality. For example, we may 
always start with DTS[n'^] for some /c, and if we derive DTS[n'^] C DTS[n'^~^] then we have a 
contradiction, and every proof that works otherwise can be rewritten to work like this. There are 
several simplifications of this type and while their proofs are not very enlightening, as a whole they 
let us identify the relevant parts of proofs. They also help us prove limitations on the proof system. 

The chart on the next page gives a graph of experimental results from a search for short proofs 
of time-space lower bounds for SAT. The x-coordinate is the number of lines in a proof (the number 
of Speedup/Slowdown applications) and the y-coordinate is the exponent of the best lower bound 
attained with that number of lines. 

Up to 25 lines, the search was completely exhaustive. Beyond that, I used a heuristic search 
that takes a queue of best-found proofs for small lengths and tries to locally improve them by 
inserting new rule applications. When a better lower bound is found, the new proof is added to 
the queue. This heuristic search was run up to about 50-|- lines. Now, all of the best proofs found 
up to 50-|- lines have a certain pattern to them, resembling the structure of the 2cos(7r/7) lower 
bound. Restricting the search to work only within this pattern, we can get annotations for 70-|- 
lines which still exhibit the pattern. Checking a 383-line proof of similar form, the lower bound 
attained was n^-^^^'^ , very close to 2cos(7r/7) ^ 1.8019. These experimental results lead us to: 
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Conjecture 3.9 The best time lower bound for SAT (in n°^^^ space) that can be proved with the 
above proof system is the ^2003(77/7) jjo^j^i of JWUOT^J- 

Given the scale at which the conjecture has been verified, I am fairly confident in its truth 
although I do not know how to prove it. Unlike [TSSWOO] , we do not know how to place a 
finite upper bound on all the parameters, namely the lengths of proofs. The conjecture is indeed 
surprising, if true. The general sentiment among researchers I have talked to (and anonymous 
referees from the past) was that a quadratic time lower bound (or more precisely, n'^~^ for all 
e > 0) should be possible with the ingredients we already have. We can show formally that a better 
lower bound than this cannot be established with the current approach. 

Theorem 3.10 In the above proof system, one cannot prove that SAT requires r? time with n°^^^ 
space algorithms. 

This theorem is proven by minimal counterexample: we take a minimum proof of a quadratic 
lower bound, and show that there is a subsequence of rules that can be removed such that the 
underlying LP remains feasible with the same parameters as before. So one possible strategy for 
proving the conjecture is to find some subsequence of rules that must arise in any optimal proof, 
and show that if one assumes c > 2cos(7r/7) then this subsequence can be removed from the proof 
without weakening it. However this strategy appears to be difficult to carry out. 

4 Finding Small Circuits 

In this last section, I will speculate about a computational approach to understanding Boolean 
circuit complexity. The following has been a joint effort with Maverick Woo. 
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Our knowledge of Boolean circuit complexity is quite poor. (For concreteness, let us concen- 
trate on circuits comprised of AND, OR, and NOT gates.) We do not know how to prove strong 
circuit lower bounds for problems in P; the best known is 5n |LR01t llM02j . One good reason why 
we don't know much about the true power of circuits is that we don't have many examples of 
minimum circuits. We don't know, for example, what an optimal circuit for 3x3 Boolean matrix 
multiplication looks like. 

It is possible that we could make progress in understanding circuits by cataloging the smallest 
circuits we know for basic functions, on small input sizes (such as n = 1, . . . , 10). This suggestion 
makes more sense for some problems than others. For SAT, the circuit complexity can depend 
on the encoding of Boolean formulas; for matrix operations, the encoding is clear. Sloane and 
Plouffe first published ^SP95j and now maintain the Online Encyclopedia of Integer Sequences, an 
exhaustive catalog of interesting sequences that arise in mathematics and the sciences. Might we 
benefit from an Encyclopedia of Minimum Circuits? For example, what do the smallest Boolean 
circuits for 10 x 10 Boolean matrix multiplication look like? Are they regular in structure? It 
is likely that the answers would give valuable insight into the complexity of the problem. The 
best algorithms we know of reduce the problem to matrix multiplication over a ring, which is then 
solved by a highly regular, recursive construction (such as Strassen's |Str69j ). Even if the cataloged 
circuits are not truly minimal but close to that, concrete examples for small inputs could be useful 
for theoreticians to mine for inspiration, or perhaps for computers to mine for patterns via machine 
learning techniques. The power of small examples cannot be underestimated. 

How can we get small examples of minimum circuits? One potential approach is to reduce this 
task to the task of developing good solvers for quantified Boolean formulas, an area in AI that 
has seen much technical progress lately. We can pose the problem of finding a small circuit as 
a quantified Boolean formula (QBF), and feed the QBF to one of many recently developed QBF 
solvers. A QBF ^s,n for size-s n x n matrix multiplication circuits can be stated roughly as: 

^s,n = (3 circuit C of s gates, 2n^ inputs, n'^ outputs) (V n x n matrices X,Y)[X - Y = C{X,Y)], 

where the predicate can be easily encoded as a SAT instance. In our encoding, we allow the circuits 
to have unbounded fan-in. For simplicity, we searched for circuits made up of only NOR gates. 

Experiments with QBF solvers have not yet revealed significant new insight. So far, they 
have discovered one fact: the optimal size circuit for 2x2 Boolean matrix multiplication is the 
obvious one. Well, duh. What about the 3x3 case? This is already difficult! The sKizzo QBF 
solver [Ben05| can prove that there is no circuit for 3x3 that has 10 gates, but nothing beyond 
that. Even when we restrict the gates to have fan-in two, the solver crashes on larger instances. 

I do not see this limited progress as a substantial deterrent. On the practical side, QBF solvers 
have only seen serious scientific attention in the last several years, and huge developmental strides 
have already been made. On the theoretical side, if we look at matrix multiplication over a field, 
a better way to approach the problem would be to phrase the QBF for small circuits as something 
like a "Merlin- Arthur formula": that is, we guess the circuit to be used and verify that it computes 
the product by evaluating on random matrices. In that case, the instances should be much easier to 
solve. In general, we can find approximately minimum circuits for problems with polysize circuits 
in ZPP'^^ |BCGKT96j . By using a high-quality SAT solver in place of the NP oracle, the idea of 
building an approximate circuit encyclopedia does not seem too implausible. However some eff'ort 
will be needed to adapt the results to work in practice. 

I do believe that in the near future, the general problem of finding small minimal circuits for P 
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problems will be within the reach of practice. Analyzing these new gadgets should inject a fresh 
dose of ideas in the area of circuit complexity. 
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